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Abstract 

In this paper we present the resalts of an experimental study uader- 
taken to assess the improvement in program quality by using formal spec- 
ifications. Specifications in the Z notation were developed for a simple 
bat realistic anti-missile system. These specifications were then used to 
develop 2 versions in C by 2 programmers. Another set of 3 versions in 
Ada were independently developed from informal specifications in Engfish- 
A comparison of the reliability and complexity of the resalting programs 
suggests the advantages of using formal specifications in terms of number 
of errors detected and fault avoidance. 
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EXTENDED ABSTRACT 


Specification languages are widely accepted as a stepping stone for design 
and development of a complex software system [1, 2, 3, 4, 5]. The advantages of a 
specification language are often not immediately clear in terms of program qual- 
ity and reliability. Proving an executable program correct for complex systems 
is computationally an intractable task [6]. Also ”an effective testing strategy 
which is reliable for all programs cannot be -.onstructed” [7], In such a setting, 
formal specification languages coupled with structured design methodologies [8] 
provide a streamlined approach for software design and development. 

In this experimental investigation, we study the effect of the specification 
language Z [11] on program reliability and complexity. For our experiment we 
chose the NASA Launch Interceptor Problem(LIP) since it has been used ex- 
tensively for several other studies in software reliability and fault tolerance. It 
is a simple but realistic representation of an anti-missile system. The original 
specifications were taken from Knight and Leveson [12]. The LIP is a constraint 
satisfaction problem, a solution to which is a decision procedure which takes a 
set of input points and launch characteristics to evaluate a set of initial launch 
conditions, called the preliminary unlocking matrix. The procedure then eval- 
uates a logical combination(the combination is decided by an input matrix) of 
the initial conditions called the final unlocking vector the components of which 
collectively decide if the launch signal should be true or otherwise. 

The experiment consisted of usual phases of software design and develop- 
ment with minor differences. In the specification phase a set of specificaaons of 
the requirements was developed in the Z notation and was validated by other 
specifiers. Several versions were developed based. pn informal and formal require- 
ments specifications separately, by independent groups of programmers. For 
testing, a hybrid approach [13] was developed based on functional and struc- 
tural information about the LIP. For generating rest cases, the hypothetical 
launch conditions were divided into 7 relatively independent groups. The truth 
values of one of the groups was fixed a priori, and* an input data set was con- 
structed to satisfy the prefixed truth values of t*' .s group and the truth values 
of the rest wer- computed against the input set. Such manually designed test 
cases were used to test each program. Ai" debugging, when the computations 
of launch conditions for all the versions match, the cyclomatic complexity mea- 
sure [9] is appli<*d to compute internal complexity of each individual module. 
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Also computed are the external complexity due to the interconnections between 
various modules based on "information flow” concepts [10], and finally the total 
system complexity as a weighted sum of internal and external complexities. 

The versions based on informal requirements are found to be afflicted with 
usual problems caused by the inherent ambiguities in the informal requirements. 
However, a significant reduction was observed in the number of errors detected 
in the testing phase in case of the versions based on formal requirements. Fur- 
ther, complexity measures strongly suggest that versions based on formal spec- 
ifications are less complex and more reliable than these based on informal re- 
quirements. The study also suggests that the formal specifications developed 
through several successive stages of operations refinement lend themselves to an 
automatic modular program development(special case of a divide and conquer 
technique) in an optimal way, and thus reduce the error-proneness of the pro- 
gram and malre it more reliable. 

Summary of Experimental Results 

L Productivity: 



Table 3 - Number of errors detected during development 
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Table 4 • Number of errors detected daring testing 


Version number 

Total Number of Errors 

Cver I 

0 

Cver II 

7 

Adaver I 

13 

Adaver II 

11 

Adaver III 

8 
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OUTLINE 


• Objectives of Study 

• Experimental Appraoch 

• Results of Experiment 

• Comparison with Versions from Informal Specifications 

• Fault Aviodance by Using Z 
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OUTLINE 


• Objectives of S tody 


• Experimental Appraoch 

- LIP Problem 

- Z-Spedfications 

• Experiment Description 


• Results of Experiment 

- Development Effort 

■ Size and Complexity Metrics 

- Errors During Development 

- Errors During "Operational Testing" 


• Comparison with Versions from Informal Specifications 
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OBJECTIVES OF STUDY 




• Investigate the effect of using formal specifications on 

- productivity 

- reliability 

- complexity 


• Compare results with versions developed from informal 
specifications 
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EXPERIMENTAL APPRAOCH 
Used NASA - Launch Interceptor Problem (LIP) 


Developed Z-specifications from English specifications of 
LIP (Two independent Z specifications) 


Used Z-specs to develop 3 indipendent versions In C 


Each version tested for a set of 54 test cases from a previous 
experiment involving LIP 


Each version executed for one million random test cases to 
simulate operational testing 



LIP 


• Simple, but realistic anti-missile system. 


• Studied elsewhere* in connection with fault-tolerant and 
Fortran/Ada comparison software research 


• Program reads inputs which represent radar reflections, 
checks whether some prespecifled conditions are met and 
determines if the reflections come from an object that is a 
threat and if yes, signals a launch decision 


* Knight and Leveson, IEEE-TSE, January 1986. 
Goel, etai, COMPSAC 87 and RADC-TR-88-213. 


A. God 
Syracs* Uv 
PatcllaT 2S 












EXAMPLE 


Launch Interceptor Conditions 


LIC 1: There exists at least one set of two consecutive data 
points that are a distance greater than LENGTH 1 
apart 


0 < = LENGTH1 


LIC 11: There exists at least one set of three data points 
separated by exactly E and F consecutive 
intervening points, respectively, that are the vertices 
of a triangle with area greater than AREA1 

1<E 

1 < F 

E + F < NUMPOINTS - 3 
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Z-SPECIFICATION LANGUAGE 


Well known specification language developed by 
Programming Research Group at Oxford University 


Has been applied to develop specifications for several 
software systems but we are not aware of experimental 
results comparing it with informal approaches 
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SOME COMMENTS ON Z FOR LIP 


Z-specifications were helpful in several aspects. 

Some Examples: 

• In resolving certain ambiguous issues 

• whether two identical (x, y) pairs can belong to a sequence 
of input data points 


• In expressing invariant properties 

• the LCM matrix is symmetric, can be easily expressed 
mathematically 


• In exploiting the repetitiveness of certain launch conditions 
which was helpful in functional groupings for design and 
testing. 

- a closer look at LIC 1, 8 and 13 indicates that they are 
related. We exploit the similarity by defining a 
"prototype” schema, and then using it to define each of 
these separately 
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Expressing Requirements in the Z 

Notation 

Example : LIC 1 


» u 


Informal Specification 

LIC 1: There exists at least one set of two consecutive 
data points that are a distance greater than LENGTHl apart. 
(LENGTH1 >0) 

Formal Specification 


LICl[NUM POINTS, LENGTHl] 

POINTS : seq Rx R 

cmv, cmv' : jV — * B 

cmv' — emu© 

{1 ~ (1 < #{{POINTS(i),POINTSU))\Vi,j : 1 ..NUM POINTS* 
j = » + 1 A (LENGTHl < cdist(POINTS(i),POINTS(j)))} 
ALENGTH 1 > 0)} 

where edist(p,q ) computes the distance between points p 
and q. 


0 - 6 - 
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Expressing Requirements in the Z 

Notation 

Example:LIC7 

Informal Specification 

LIC 7: There exists at least one set of N-PTS consecutive 
data points such that at least one of the points lies a dis- 
tance greater than DIST from the line joining the first and 
last of these points. If the first and last points of these N_PTS 
points are identical, then the calculated distance to compare 
with DIST will be the distance from the coincident point to all 
other points of the N.PTS consecutive points. (DIST >0) 

Formal Specification 


LIC"[NU M POINT S,NpTS, DIST 
POINTS : seq Rx R 

cmv , cmv ' : N-+ 8 

cmv' = cmv © 

{7 ~ (1 < #{(POINTS(i),POINTS(j))\Vi,j : 1..NUM POINTS* 
j = i -b N-PTS — 1 A 3 k: i + l..j — 1* 

(pt _cmp(PO I NT S{i), POINTS(j )) 
A{cdist(POINTS[i),POINTS(k)) > DIST)) 
V(-<pt-cmp(POINTS(i), POINTS(j)) 

/ \{pdist{POINTS{i),POINTS(j),POINTS{k )) > DIST))} 

ADI ST > 0 )} 

where edist(p, q) computes the distance between points p 
and q , pdist(p, q, r) computes the perpendicular distance from 
point r to the line through p and q and pt-cmp(p , q) returns a 
boolean value true if p and q are identical, and otherwise false 
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Expressing Requirements in the Z 
Notation(contcL) 

Example:LIC7 


pdist : ft 2 x Tl 7 x V? -+ TV 

Note that the line must be well defined, i.e, at least the points 
on the line must not be identical. Obviously this is a partial 
function. 
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RESULTS OF EXPERIMENT 
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SOME PROGRAM METRICS 


Programmer 

C-Code From 
2 -Specs 

Ada Code From 
Informal Specs 

A 

Ft 

C 

D 

E 

F 

Source Lines 

373 

407 

669 

691 

624 

851 

Comment Lines 

82 

80 

59 

59 

126 

251 

System Complexity* 

56 

53 

81 

334 

309 

297 


* See Lew et al, TSE, November 1988. 
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COMPLEXITY METRIC 


• System S has n modules, each with complexity M| 


• System complexity = VlMj 

• Mi depends on 

• Internal complexity 

- External complexity (measures module interrelationships) 

• Internal complexity 

- McCabe’s cydomatic number 


• External complexity 

* Amount of interaction with the environment 
- Depends on the information content of input and output 
parameters 


* Lew et al, IEEE-TSE, November 1988, pp. 1645-1655. 
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PROGRAM DEVELOPMENT EFFORT (hours) 


Versions 

Develop 

Z-Specs 

Design 

Coding 

Testing 

Total 

A 

27 

6 

6 

6 

45 

B 

10* 

10 

10 

8 

38 

C 

33 

8 

6 

4 

51 


* B used specs, developed by A 


Learning Z: A - 20 hrs. 

C - 21 hrs. 
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NUMBER OF ERRORS 
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COMPARISON OF DATA FROM C AND ADA VERSIONS 


• We compared the effort and error data from a previous 
experiment that used Fortran and Ada languages. 


• We do not think that our results are biased because 
language dependent aspects are not under study here. Also, 
the programmers in these studies were reasonably proficient 
in the respective languages so that the choice of the language 
should not affect our results 


• However, to enhance our conclusions, we plan to develop C 
versions from informal specifications 
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COMPARISON OF EXPERIMENTAL RESULTS: EFFORT AND ERRORS 




D&UT - Development and Unit Testing 
FT - Function Testing 



i 
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FAULT AVOIDANCE BY USING Z 


• We believe that certain types of faults can be avoided by 
using formal specifications 


• Following are two explicit examples of faults avoided by 
using for LIP* 

- Caiuclation of angle between x and 2 x rather than between 
Oandx 

- Calculation of distance from point to line when points are 

collinear and first point not between other two 


* See Brilliant et al, TSE, February 1990, page 242. 
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FAULT-A VOEDANCE - EXAMPLE LIC 7 


Consider 3 collinear points (A, B, C) as shown 


B 


• Need to compute distance from B to line AC (LIC 7) 


• Computation* from informal specs can lead to 
Dist(A, C, B) = min (dist(A, B), dist(B,C» 


# However, formal specifications always compute zero, the correct 
result 


See Brilliant et al, TSE, February 1990, p. 242. 
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CONCLUDING REMARKS 


• Use of Z specifications was clearly helpful in reducing errors 
(and hence increasing reliability) 


• Based on a few metrics, it is also evident that the complexity 
of code developed from Z was also lower 


• Total effort involved, including learning Z and development 
of Formal specifications, was comparable to the effort 
involved in developing versions from informal specifications 


Yet — 


• This experiment does not provide conclusive evidence about 
the superiority of formal specification over informal ones 


• Further investigation necessary to explore the feasibility and 
usefulness of Z for large problems 


• Reusability of such formal specifications also needs to be 
investigated 
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